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FOREWORD 


The  Jet  Propulsion  Laboratory,  Pasadena,  California,  assisted  in  the 
preparation  of  this  document  under  Contract  Number  DAEA  18-82-F-3551 , 
Modification  04  dated  September  22,  1982. 


SECTION  1 .  SUMMARY 


1.1  BACKGROUND 

a.  The  Army  and  the  Department  of  Defense  (DOD)  have  developed  and  are 
continuing  to  develop  a  number  of  major  automated  Command  Control,  Com¬ 
munications,  and  Intelligence  (C3I)  Systems.  These  include  TACFIRE,  MCS , 

TCAC ,  TCS/TCT ,  AN/TSQ-73,  REMBASS,  and  ASAS  to  name  a  few.  These  systems  are 
designed  to  interface  with  a  large  number  of  interactive  systems  and  to  oper¬ 
ate  in  a  highly  interactive  environment.  A  critical  element  in  the  success  or 
failure  of  these  C3I  systems  will  be  their  ability  to  interoperate  arid  perform 
under  load  in  a  highly  interactive  tactical  environment. 

b.  The  nature  of  interoperability  testing  is  such  that  large  volumes  of 
data  are  generated.  This,  in  turn,  leads  to  the  requirement  for  an  automated 
data  reduction  and  analysis  capability  since  the  cost  of  manual  data  reduction 
and  analysis  in  terms  of  both  dollar?  and  test  personnel  is  prohibitive. 
Additionally,  the  probability  of  errors  in  manual  data  reduction  and  analysis 
further  supports  this  requirement.  The  Army  is  developing  major  systems  with 
a  critical  role  in  the  combat  effectiveness  of  the  future  Army  in  the  field. 
The  possibility  of  failure  to  analyze  and  evaluate  the  data  generated  in 
interoperability  testing  is  a  risk  that  the  Army  cannot  afford  to  take. 

c.  The  Interim  Test  Item  Stimulator  (ITIS)  is  a  message  loading  system 
used  by  the  US  Army  Electronic  Proving  Ground  (USAEPG)  for  development  testing 
(DT)  of  these  automated  Army  systems.  The  requirements  for  the  ITIS  were 
articulated  in  mid  1979  and  a  generation  of  software  to  meet  those  require¬ 
ments  was  begun  at  that  time.  The  ITIS  consists  of  three  distinct  parts;  the 
pre-test  functions,  real-time  functions,  and  post-test  analysis  functions. 
However,  the  ITIS  does  not  provide  a  truly  automated  post-test  analysis 
capability . 

d.  This  report  provides  a  synopsis  of  the  known  efforts  toward  auto¬ 
mating  the  data  reduction  and  analysis  of  interoperability  test  data. 

1.2  OBJECTIVE 

The  objective  was  to  develop  comprehensive  end  cost  effective  methods  for 
reducing  and  analyzing  the  large  volume  of  test  data  generated  during  inter¬ 
operability  testing  of  automated  communica^ions'-electronicc  (C-E)  systems. 
Specifically,  two  sub-objectives  are  addressed: 

a.  To  determine  current  tester  needs  for  automation  of  interoperability 
post-test  data  analysis. 

b.  To  propose  an  appropriate  approach  for  the  automation  of  inter¬ 
operability  post-test  data  analysis  which  would  provide  an  effective  method 
for  reduction  and  analysis  of  typical  test  data  from  a  large  data  base 
generated  during  interoperability  testing. 

1.3  SUMMARY  OF  PROCEDURES 

a.  The  procedures  used  to  identify,  isolate,  and  resolve  technical 
problems  in  the  course  of  performing  this  investigation  included  interviews 


1-1 


with  US  Army  test  personnel,  reviews  of  pertinent  documents  and  contacts  with 
outside  vendors.  The  scope  was  limited  to  examining  only  a  select  number  of 
systems  with  emphasis  on  the  Maneuver  Control  System  (MCS)  since  that  system 
is  an  executive  system  within  the  Army  Command  and  Control  System  (ACCS) 
architecture;  USAEPG  has  familiarity  with  the  MCS  owing  to  recent  tests. 

b.  Interviews  were  conducted  with  personnel  a;  the  Field  Artillery 
Tactical  Data  System  (FATDS)  Software  Support  Group  (SSG)  at  Fort  Sill, 
Oklahoma  to  determine  present  post-test  data  analysts  capabilities  at  FATDS 
SSG,  and  to  identify  and  acquire  applicable  documents. 

c.  MCS  specifications,  Required  Operational  Capability  (ROC)  documents, 
relevant  Independent  Evaluation  Plans  (IEP),  and  Test  Design  Plans  (TDP)  were 
reviewed  to  determine  interoperability  and  other  test  requirements  for  the  MCS 
as  well  as  post-test  data  analysis  requirements. 

d.  Contacts  with  outside  vendors  included  presentations  by  companies 
supplying  state-of-the-art  relational  data  base  systems  and  suppliers  of  air 
defense  test  hardware  and  software.  The  test  hardware  and  software  included 
real-time,  quick-look,  and  post-test  analysis  capabilities. 

1.4  SUMMARY  OF  RESULTS 

a.  The  Army  Battlefield  Interface  Concept  (ABIC)  identifies  the  inter¬ 
faces  for  each  Army  system.  For  MCS  alone,  there  will  be  numerous  inter¬ 
operability  tests  performed  in  the  next  three  to  fojr  years.  The  MCS  must  be 
able  to  interoperate  successfully  with  up  to  fifteen  other  systems.  MCS 
interoperability  testing  must  be  done  for  multiple  levels  of  loading  ranging 
from  no  load  up  to  worst  case  loading.  Data  item  requirements  include: 


(1) 

data  on  transmission  errors 

(2) 

format  errors 

(3) 

transmission  delays 

(4) 

central  processing  unit  (CPU)  demands 

(5) 

message  volume 

(6) 

translation/reformat  times 

system 

(7) 

response  times  for  information  requests  for  each  interoperating 

b.  Post-test  analysis  performed  at  the  FATDS  SSG  was  found  to  be  -basi¬ 
cally  manual  with  some  automation  in  their  single  thread,  single  function 
tests.  The  single  thread,  single  function  tests  consisted  of  sending  the 
software  under  test  each  TACFIRE  message,  one  at  a  time,  and  checking  to  see 
if  the  correct  response  was  received  (in  many  cases  the  correct  response  was  ; 
simple  acknowledgement). 
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c.  Review  of  the  present  ITIS  functions  and  procedures  revealed  two 
specific  weaknesses  with  regard  to  automation.  First,  the  post-test  analysis 
system  is  labor  intensive,  requiring  detailed  planning  and  a  large  number  of 
inputs  by  an  operator  for  each  analysis  product  (output)  requested.  Second, 
the  message  scenario  tape  (MST)  generation  is  also  labor  intensive.  The  MST 
generation  relies  on  an  ITIS  unique  data  base,  and  there  is  no  direct  tie  be¬ 
tween  the  MST,  the  message  log  tape  (MLT),  and  post-test  function.  Of  concern 
here  is  the  relation  of  the  MST  to  the  post-test  function,  that  is,  did  the 
scenario  execute  as  planned? 

d.  Current  tester  needs  are  summarized  as  follows: 

(1)  Quickly  and  easily  be  able  to  define  the  data  and  enter  it  into 
a  computer. 

(2)  Quickly  and  easily  retrieve  data  based  on  selected  filters 
(attributes)  . 

(3)  Quickly  and  easily  be  able  to  define  and  output  the  required 
graphs  and  charts. 

(4)  Be  able  to  automatically  conduct  statistical  analyses  on  the 
outputted  information. 

(5)  Be  able  to  compare  results  from  different  tests;  also,  be  able 
to  compare  results  from  tests  performed  at  one  point  in  time  with  version  "x" 
software  to  tests  performed  at  another  point  in  time  with  version  ”y" 
software . 

(6)  Quickly  and  easily  verify  that  the  scenario  was  executed  as 
planned  or  identify  any  differences  if  executed  differently  from  planned. 

e.  Two  approaches  for  the  automation  of  interoperability  post-test  data 
analysis  are  proposed.  The  first  approach  uses  the  current  post-test  analysis 
system  (PTAS)  software  to  the  maximum  extent,  along  with  a  pre-defined 
runstream  (in  VAX  terminology,  a  command  file)  to  automatically  generate  the 
desired  product  (output).  The  runstream  is  a  set  of  computer  commands  which 
replaces  the  normal  (redundant)  PTAS  operator  entries.  The  second  approach  is 
predicated  on  the  introduction  of  a  commercial ly  available  data  base 
management  system  (DBMS)  which  includes  a  query  and  embedded  query  language. 
The  query  language  is  used  in  place  of  the  PTAS  software  to  do  the  aata 
retrieval . 

1.5  DISCUSSION 

a.  Interoperability  testing,  as  experienced  with  MCS,  has  produced  large 
quantities  of  data  requiring  analysis  and  reporting.  No  relief  on  the 
voluminous  data  obtained  is  foreseen  as  more  and  more  systems  are  linked 
together  and  interoperability  testing  continues.  Automation  of  test  data 
analysis  must  be  pursued  to  effectively  utilize  and  analyze  the  voluminous 
data.  The  ITIS  can  provide  information  on  transmission  errors,  format  errors, 
transmission  delays  [as  seen  by  the  ITIS  at  the  3ystem  under  test  (SUT)-ITIS 
interface],  message  volume,  translation/reformat  times  (SUT-ITIS  interface), 
and  response  times  (SUT-ITIS  interface).  To  obtain  information  about  CPU 
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;  demands,  software  instrumentation  other  than  the  ITIS,  such  as  hardware  or 

•  software  monitors,  is  required.  The  added  instrumentation  could  also  be  used 

if  intrinsic  information  on  transmission  delays,  translaticn/reformat  times, 
j  and  response  times  are  required  to  isolate  the  cause  of  any  delays  or  bottle- 

j  necks  within  the  SUT.  The  ITIS  PTAS  does  not  currently  have  the  capability  to 

.  accept  information  from  additional  data  sources  for  correlation  and  analysis. 

Such  a  capability  would  further  enhance  ITIS  for  automation  of  test  data 

.  analysis. 

i 

,  b.  The  method  used  by  the  FATDS  SSG,  that  is  to  identify  the  correct 

I  response  from  a  3UT  given  a  particular  message  was  inputted,  is  another  means 

for  automation  of  test  data  analysis.  If  a  SUT  is  working  reasonably  well, 

!  the  amount  of  "good"  data  received  may  bury  those  anomalous  data;  this  method 

,  would  aid  in  sorting  and  identifying  those  anomalous  cases.  The  method  needs 

'  to  be  expanded  to  handle  multi-thread,  multi-function  tests.  TACFIRE,  for 

.  example,  is  a  case  where  the  correct  response  to  a  given  message  stimulus  is 

I  dependent  upon  configuration. 

c.  The  ITIS  PTAS  is  a  step  towards  automation;  however,  it  is  labor 
intensive.  The  PTAS  software  is  very  flexible  as  to  selection  of  inputs  and 
outputs,  but  this  flexibility  means  that  a  large  effort  is  required  to  define 

j  the  input  tables  for  each  analysis  product  desired. 

d.  Items  2  through  5  of  the  list  of  tester  needs  can  be  accomplished  by 
use  of  a  generalized  data  base  management  system  (GDBMS)  and  is  further 

'  discussed  in  paragraph  1.5e.  Item  5  can  utilize  a  GDBMS  if  measures  of 

performance  (MOPs)  are  defined  and  are  part  of  the  data  base;  hence,  any 
!  performance  difference  due  to  software  version  changes  can  be  examined  by 

j  comparing  the  pre-defined  MCPs.  Items  1  and  6  need  not  use  a  GDBMS;  user 

specified/generated  programs  can  be  used  and  tailored  for  specific  SUT 
analyses  and  scenario  design.  Operating  system  utilities,  such  as  the  compare 
program  on  the  VAX,  could  be  used  ro  determine  differences  from  a  planned 
scenario  to  that  actually  executed. 

e.  The  use  of  a  runstream  to  drive  existing  software  as  a  proven  method 
of  automation  is  relatively  straightforward  and  low  cost  to  implement,  as  in 
this  case  where  PTAS  already  Is  in  existence.  Although  the  runstream  auto¬ 
mates  the  manual  procedures  normally  performed,  changes  to  the  desired  output 
are  not  easily  and  quickly  performed. 

f.  The  use  of  a  query  and  embedded  query  is  a  natural  extension  of  the 
modern  commercial  data  base  systems.  It  provides  for  tabularization  and  other 
required  data  operations;  the  programming  can  be  done  by  persons  familiar  with 
high  order  language  programming.  The  test  analyst  does  not  have  to  define 
procedures  when  changes  of  outputs  are  required;  the  DBMS  does  it  automati¬ 
cally,  with  the  analyst  specifying  what  ha  needs.  Additionally,  modern  day 
DBMSs  include  data  analysis/graphics  packages,  thereby  alleviating  the  need  to 
create  a  file  for  interfacing  with  yet  another  analysis  package  (PTAS  inter¬ 
faces  with  the  DBMS  statistical  package).  The  use  of  DBMS  with  a  query 
language  is  the  logical  evolution  of  the  ITIS  system. 


1-4 


N.*> 


1.6  CONCLUSIONS 


a.  There  are  no  known  completely  satisfactory  automated  analysis  systems 
for  interoperability  test  data.  Such  a  system  is  greatly  needed  to  provide 
comprehensive  and  cost  effective  methods  for  reducing  and  analyzing  the  large 
volume  of  test  data  generated  during  interoperability  testing  of  automated  C-E 
systems . 


b.  The  best  approach  toward  automating  interoperability  post-test  data 
analysis  is  the  implementation  of  a  GDBMS.  This  allows  greater  flexibility 
and  capability  than  afforded  by  currently  used  interoperability  data  analysis 
means . 

1.7  RECOMMENDATION 

Use  of  a  GDBMS  should  be  pursued  for  use  in  automated  post-test  data 
reduction  and  analysis  of  interoperability  test  data. 
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2.1  INTRODUCTION 

a.  Two  objectives  to  be  considered  when  doing  this  methodology  investi¬ 
gation  are  to  determine  current  MCS  user  needs  for  automation  of  inter¬ 
operability  post-test  da  inalysis,  and  to  propose  an  appropriate  approach 
for  the  automation  of  in  oerability  post-test  data  analysis  for  the  MCS 
which  would  provide  an  ef-.ctive  method  for  reduction  and  analysis  of  typical 
test  data  from  a  large  data  base  generated  during  interoperability  testing. 

b.  There  are  numerous  definitions  of  interoperability,  of  which  two  are 
given  as  follows. 

(1)  Interoperability  (ref  1,  app  C):  The  capability  to  interact 
with  existing  and  future  information  systems  of  ground  combat,  other  services, 
and  with  the  command  and  control  systems  of  allied  nations. 

(2)  Interoperability  (ref  2,  app  C):  The  ability  of  systems,  units, 
or  forces  to  provide  services  to  and  accept  services  from  other  systems, 
units,  or  forces  and  to  use  the  services  so  exchanged  to  enable  them  to 
operate  effectively  together. 

c.  Interoperability  testing  by  the  US  Army  is  designed  to  evaluate  the 
following  elements  (ref  2,  app  C): 

(1)  Message  transfer  compatibility,  accuracy  and  timeliness. 

(2)  Adequacy  of  Table  of  Organization/Equipment  (TOE)  communications 
means  to  support  message  transfer  between  two  systems,  including  communication 
net  saturation  during  interoperability  and  non-interoperability;  and  adequacy 
of  authorized  equipment  to  satisfy  interoperability  criteria. 

(3)  Effects  of  noise  from  battlefield  conditions,  where  noise  is 
defined  to  be  unpredictable  electromagnetic  radiation  due  to  components, 
atmosphere,  unintended  systems  coupling,  or  electronic  countermeasure/ 
electronic  counter-countermeasures  (ECM/ECCM)  activity. 

(4)  Adequacy  of  operator  training  in  effecting  message  transfer  be¬ 
tween  systems. 

(5)  The  detectability  rate  of  two  systems  interoperating  together. 
Detectability  here  and  in  item  (8)  means  the  ability  to  discover  or  find  elec¬ 
tromagnetic  emanations  and  to  extract  information  from  them. 

(6)  The  effect  of  data  transfer  messages  on  system  processing  time 
between  two  systems. 

(7)  The  procedures  that  affect  continuity  of  operation  between  two 
systems  in  the  event  of  failure  or  shutdown  of  the  primary  means  of  inter¬ 
operability. 

(8)  The  functional  utility  of  system-to-systera  data  transfer  on  the 
basis  of  data  communication  enhancements,  costs,  detectability,  and  other 
trade-off  factors. 
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d.  Testing  can  be  done  in  a  number  of  test  stimuli  SUT  configurations. 
Four  basic  configurations  are:  single  system,  single  threaded;  single  system, 
multiple  threaded;  multiple  system,  single  threaded;  and  multiple  system, 
multiple  threaded.  These  four  configurations  are  shown  functionally  in 
figure  1  (ref  3,  app  C). 

e.  One  function  of  DT  is  to  determine  how  well  the  SUT  meets  its 
performance  specifications.  Test  plans  are  developed  cm  the  basis  of  the 
governing  documents,  including  the  ROC,  Test  Operation  Procedures  (TOP),  the 
system  A,  B,  and  C  level  specifications,  the  IEP,  and  the  TDP.  After  a  test 
has  been  run,  the  test  data  are  evaluated  to  produce  indicators  of  system 
performance;  data  are  summarized  in  descriptive  write-ups,  tabular  form,  time 
histories,  histograms,  and  other  means  as  required.  After  all  testing  is  com¬ 
pleted,  a  final  report  is  issued  to  summarize  system  performance,  including 
performance  meeting  or  exceeding  requirements,  performance  shortcomings,  and 
recommendations . 

f.  This  section  presents  the  detailed  information  used  to  support  the 
conclusions  and  recommendations  of  the  summary  section  (Section  1).  This  sec¬ 
tion  includes  information  on  current  and  proposed  methods  of  post-test  data 
analysis  as  background  material,  identification  of  MCS  user  needs  for  auto¬ 
mation  of  interoperability  post-test  data  analysis  using  mainly  the  currently 
available  software  and  hardware;  and  a  proposed  approach  in  the  event  of  a 
major  change  to  a  newer  commercially  available  D3MS. 
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SUT 


d.  Multiple  systems,  multiple  threaded 
Figure  1.  Interoperability  test  configurations. 
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2.2  METHODS  OF  POST-TEST  ANALYSIS 


In  a  consideration  of  automating  post-test  data  analysis,  one  would 
naturally  ask  how  it's  being  done  currently  at  USAEPG,  how  it's  being  done  on 
other  US  Army  and  military  programs,  and  how,  it's  being  done  on  non-military 
systems  or  programs,  e.g.,  at  the  Jet  Propulsion  Laboratory  (JPL).  The  fol¬ 
lowing  information  was  gathered  during  two  trips  to  Fort  Huachuca  (ITIS/MCS 
testing),  one  trip  to  Fort  Sill  (TACFIRE  testing),  and  one  trip  to  a  vendor 
involved  with  testing  of  air  defense  systems. 

a.  ITIS/MCS  Post-Test  Data  Analysis 

(1)  The  MCS  interoperability  testing  is  done  using  the  USAEPG  ITIS. 
Figure  2  shows  the  functional  configuration  of  the  ITIS.  As  shown,  there  are 
three  main  functions:  the  pre-test  system,  which  generates  the  message 
scenario  tape  (MST);  the  real-time  system,  which  accepts  inputs  from  the  MST, 
exercises  the  SUT,  and  stores  the  test  data  to  the  log  tape;  and  the  post-test 
analysis  system,  which  is  used  to  analyze  the  data  on  the  log  tape.  PTAS  is 
the  system  of  interest  here. 

(2)  An  earlier  version  of  ITIS  was  to  have  had  both  the  MST  and  log 
tapes  as  inputs  to  the  PTAS,  but  the  current  version  of  the  test  system  has 
the  MST  information  included  on  the  log  tape  and  no  direct  input  from  the  MST 
to  the  PTAS.  This  method  should  require  that  the  MST  information  to  be 
relayed  directly  through  the  real-time  system  to  the  log  tape  with  no  changes 
due  to  SUT-caused  message  delays  or  errors,  else  it  becomes  difficult  or 
impossible  to  do  valid  evaluations  of  SUT  performance,  or  to  verify  whether 
the  scenario  went  off  as  planned. 

(3)  The  current  version  of  PTAS  operates  as  follows.  The  PTAS  user 
controls  reduction  of  the  data  by  entering  control  parameters  at  a  terminal. 
The  PTAS  produces  printed  output  in  the  form  of  dumps,  tables,  or  plots,  as 
requested;  the  system  can  also  save  the  output  to  disk  files  to  be  used  as 
input  to  a  statistical  program  package.  This  package,  which  is  not  part  of 
the  PTAS,  can  be  used  to  generate  histograms  and  other  statistical  outputs.  A 
functional  representation  of  the  post-test  analysis  system  is  shown  in 
figure  3. 


(4)  The  PTAS  is  run  on  a  VAX  11/780  with  the  following  peripherals: 
terminal,  line  printer,  mag  tape  transport,  and  disk  storage.  The  analysis  of 
data  by  the  PTAS  i3  controlled  by  the  specification  of  inputs  for  up  to  five 
control  tables:  format,  primary  variable  or  variable,  dump,  table,  and  plot. 
These  tables  control  the  data  reduction  and  specify  the  output  formats.  An 
edit  function  allows  the  user  to  add  or  delete  table  entries.  A  functional 
representation  of  the  control  table  operation  for  PTAS  is  shown  in  figure  4. 
Further  details  on  PTAS  usage  are  contained  in  the  PTAS  User's  Manual  (ref  5, 
app  C),  and  are  discussed  in  paragraph  2.3b(2). 
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Figure  A.  Internal  data  flow  of  the  PTAS  program. 


b.  TACFIRE  Post-Test  Data  Analysis 


(1)  TACFIRE  is  a  US  Army  command  and  control  system  that  is  cur¬ 
rently  being  deployed  with  combat  units-  This  system  receives,  stores,  com¬ 
bines,  and  sorts  target  reports;  receives  targeting  information;  allocates 
firepower;  computes  ballistic  firing  data;  and  sends  fire  orders  to  field 
artillery  weapons.  TACFIREs  communications  capabilities  include  conversion  of 
standard  field  artillery  messages  to  digital  message-!  that  are  transmitted 
over  standard  Army  communication  equipment,  automatic  encryption  and  decryp¬ 
tion  of  messages,  and  automatic  relay  of  messages  (refs  6  and  7,  app  C). 

(2)  The  TACFIRE/FATDS  Software  Support  Group  (TSSG)  at  Fort  Sill  has 
responsibility  for  verification  testing  of  post  deployment  -ersions  of  TACFIRE 
field  systems.  Verification  testing  of  TACFIRE  software  programs  is  done  to 
verify  that  required  functional  changes  have  been  implemented,  that  the 
changes  meet  specified  requirements,  and  that  their  implementation  has  not 
adversely  affected  the  unchanged  portions.  Verification  testing  is  'one  in 
four  phases:  Standard  Operating  System  (SOS)  test;  Functional  Area  System 
Test  (FAST);  Benchmark  Test;  and  New  Version  Verification  Test  (NVVT) .  TSSG 
analyzes  the  test  data  off-line  following  completion  of  each  test  phase. 

(3)  Analysis  of  the  test  results  is  accomplished  manually  by  visual 
observation  during  the  conduct  of  the  test  and  by  comparing  printouts  of 
errors/warnings,  the  transaction  journal,  and  message  traffic  with  expected 
outputs;  and  automatically  by  use  of  the  TSSG  FAST  Compare  program.  Also, 
shortcomings,  deficiencies,  or  unexpected  results  discovered  during  the  con¬ 
duct  or  analysis  of  any  test  are  recorded  on  a  Test  Anomaly  Report  (TAR)  and 
forwarded  to  the  test  director  for  resolution.  Anomalies  which  cannot  be 
resolved  during  the  conduct  of  the  test  are  recorded  as  TACFIRE  Problem 
Reports  (TPRs)  for  post-test  resolution  (ref  8,  app  C). 

c.  Position  Location  Reporting  System  (PLRS) 

(1)  PLRS  Is  a  computer-based  system  that  provides  position  location 
and  navigation  Information  to  a  community  of  users  and  to  the  command  and  con¬ 
trol  elements  of  that  community.  It  provides  accurate,  real-time,  three- 
dimensional  position  location  information  of  PLRS  equipped  airborne  and  ground 
elements.  The  system  Includes  a  limited  digital  message  capability  (refs  9 
and  10,  app  C) . 

(2)  The  references  just  cited  are  the  PLRS  DT  II  Final  Report  and  DT 
II  Software  Test,  Final  Report,  Supplement  2.  These  two  volumes  are  cited 
here  as  examples  of  post-teat  data  analysis  and  reporting  that  are  done  for 
systems  development  tests.  The  finrl  report  contains  630  pages  in  the  body  of 
the  report,  plus  an  additional  610  pages  in  the  12  appendices;  supplement  2 
contains  176  pages  ir.  the  body  of  the  report  plus  an  additional  42  pages  in 
four  appendices.  There  is  considerable  narrative  concerning  the  test 
procedures  and  system  performance,  but  there  is  also  a  considerable  amount  of 
space  devoted  to  presentation  of  data  in  various  formats,  Including  plots  and 
tables.  Table  I  is  a  summary  of  the  various  means  of  data  presentation  in  the 
body  of  the  two  reports  (appendices  not  included). 
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TABLE  I.  DATA  PRESENTATION  IN  PLRS  FINAL  REPORTS 
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— 

Data 

Data 

Data 

Plot, 

Plot, 

Plot, 

Table, 

Table, 

Table , 

Printer 

Plotter 

Artwork 

Printer 

Typed 

Test  Data 

Final  Report 

1 

108 

37 

18 

32 

190 

2 

Supplement  2 

18 

24 

0 

2 

87 

0 

Totals 

126 

61 

00 
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_ 

34 

277 
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r;j 


,-j 
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(3)  Some  examples  of  the  data  plots  and  tables  from  those  reports 
are  shown  In  figures  5  through  10.  Figure  5  is  an  example  of  a  typical  two 
parameter  printer  plot,  with  page  number,  figure  number,  and  title  typed  on. 
Figure  6  is  a  histogvim  printer  plot  with  typed  page  and  title  plus  additional 
information  on  the  tea.  conditions.  Figure  7  is  a  two  parameter  plot  done  on 
a  plotter,  again  with  title  and  page  typed  in.  Figure  8  is  a  two  parameter 
plot,  but  appears  to  have  been  done  by  a  graphics  artist  by  hand,  rather  than 
mechanically.  Figure  9  is  an  example  of  a  printer  output  of  tabular  data  used 
directly  in  the  report;  finally,  figure  10  is  typical  of  the  many  presenta¬ 
tions  of  tabular  data  that  were  done  by  typing  or  word  processor. 

(4)  As  noted,  in  most  instances,  the  page  number  and  possibly  a 
figure  number  and  title  were  typed  on  the  plot  or  table  after  figure  produc¬ 
tion.  Of  course,  page  numbers  would  be  very  hard  to  coordinate,  but  typing 
figure  numbers  and  titles  does  mean  an  extra  step  in  producing  a  report. 

(5)  The  printer  plots  are  sometimes  hard  to  read  because  of 
imperfect  originals;  also  they  are  sometimes  slightly  cryptic  and  require  a 
bit  of  study  to  determine  what  i9  being  presented;  overall,  the  quality  of 
printer  plots  is  not  as  high  as  that  done  by  plotter  or  artwork,  if  for  no 
other  reason  than  that  the  printing  is  small  and  sometimes  imperfect. 

(6)  A  gross  analysis  of  the  amount  of  automation  used  in  producing 
these  reports  indicates  that  of  205  plots,  187,  or  91  percent,  were  done  by 
printer  or  plotter.  There  were  approximately  313  tables,  of  which  only  34,  or 
11  percent,  were  produced  as  printer  output.  Therefore,  the  PLRS  reports  are 
good  examples  of  automating  the  analysis  and  presentation  of  large  amounts  of 
test  data  in  the  form  of  plots;  it  would  appear  that  presentation  of  tabular 
data  is  still  being  done  in  several  steps  -  drawing  up  of  a  table  by  the 
analyst,  and  thence  to  final  form  on  a  word  processor. 

d.  Joint  Tactical  Information  Distribution  System  (JTIDS) 

(1)  A  presentation  on  ITIS  support  to  Army  JTIDS  Class  II  Terminal 
DT/OT  testing  was  made  to  the  PLRS  JTIDS  Hybrid  (PJH)  Simulation  Working  Group 
in  December  1982.  It  was  assumed  that  testing  will  be  conducted  at  Elgin  Air 
Force  Base  in  the  time  period  June  1984  to  March  1985  and  that  testing 
requirements  would  include  repeatable  scenarios,  inputs  with  planned  errors, 
and  system  stress. 

(2)  Development  testing  and  operational  testing  both  included 
interoperability  as  an  important  test  objective.  In  summarizing  ITIS  features 
for  this  application,  two  points  were  that  software  exists,  or  can  be  modified 
at  reasonably  low  risk;  and  the  ITIS  provides  a  complete  capability  for  post¬ 
test  analysis. 

(3)  The  JTIDS  testing  is  a  probable  future  use  of  the  ITIS  and  is  an 
additional  requirement  to  be  considered  if  modifications  of  the  current  PTAS 
to  a  more  automated  system  are  undertaken. 
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Scenario  98,  run  1  *' 

Figure  6.  PLRS  plot-data  output  messages  received  by  SIMPLRS  per  22  second  interval,  scenario  9B,  run 
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Figure  9.  PLRS  tabular  data  (example  1). 
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Figure  10.  PLRS  tabular  data  (example  2). 


e.  Joint  Interface  Test  System  (JITS) 


(1)  The  Joint  Interoperability  of  Tactical  Command  and  Control 
Systems  (JINTACCS)  program  involves  testing  a  large  number  of  diverse  C3I 
systems  for  compatibility  and  interoperability.  The  JITS  is  a  JINTACCS 
program.  The  C3I  systems  are  at  geographically  dispersed  locations. 

(2)  The  JITS  is  an  integrated  system  of  many  hardware  and  software 
components.  The  major  software  functions  are  test  control,  monitoring, 
simulation,  data  collection,  and  data  reduction/analysis.  Data  collection 
includes  message  traffic,  executed  events,  and  status  data.  Data 
reduction/analysis  includes  on-line/real-time  analysis,  and  post-test 
analysis . 


(3)  A  more  detailed  breakdown  of  the  data  collection  and  data 
reduction/analysis  functions  performed  by  JITS  are  as  follows: 

(a)  Data  Collection 

Initialization  data 
2_  Test  configuration  data 
3_  Exercise  ID  and  time  reference 
A_  Executed  events 
5_  Operator  initiated  test  controls 
6_  System  status 
7_  Comm  circuit  status 
8_  Messages  exchanged  on  data  links 
9_  Test  notes 

10  Equipment  configuration 

1 1  Equipment  status 

12  Detected  event  errors  and  access  anomalies 

(b)  Data  Reduction/Analysis 

1_  TADIL  message  reduction 
2_  TADIL  message  analysis 
3_  SRN  target  event  history 
4_  Chaff  event  history 
5_  SRN  admin  point  event  history 
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6_  Sin  PU/RLI  event  history 

1_  TADIL  track  event  history 

8_  ESU  history 

9  TADIL  link  assignment 

10  Executed  events 

11  Operator  initiated  test  controls 

12  Detected  errors 

13  Link  quality  data 

14  Test  notes 

(4)  The  JITS  also  includes  teletype  (TTY)  processing  which  does 
communication  test  conduct,  communication  test  generation,  and  communication 
text  analysis.  The  third  function,  communication  text  analysis,  includes 
compatibility  and  interoperability  (C&I)  test  analysis  of  message  text  and 
structure,  and  recording  of  analysis  results  for  post-test  reports.  Some  of 
the  post-test  analysis  is  done  by  manual  selection  of  variables  and  desired 
plots/tables,  in  a  manner  similar  to  tnat  of  the  current  ITIS  PTAS  software. 

(5)  This  information  on  JITS  is  presented  as  an  example  of  the  state 
of  the  technology  of  test  data  gathering  and  analysis  as  implemented  in  the 
nonpublic  sector  (private  industry).  The  techniques  and  methodology  used  by 
this  company  can  be  used  as  a  measure  of  the  ITIS  and  Its  application  to  MCS 
and  future  systems,  particularly  in  the  areas  of  automation  of  test  data 
collection  and  data  reduction  and  analysis. 
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2.3  MCS  USER  NEEDS  FOR  AUTOMATED  INTEROPERABILITY  TEST  DATA  ANALYSIS 
(AIOPTDA) 


The  first  objective  of  the  methodology  task,  is  the  determination  of 
current  MCS  user  needs  for  automation  of  interoperability  post-test  data 
analysis.  This  is  discussed  here  in  terms  of  specified  requirements  and 
current  methods  of  MCS  testing  and  analysis. 

a.  Specification  Requirements  on  Post-Test  Analysis.  A  number  of  docu¬ 
ments  must  be  considered  when  designing  tests  to  determine  the  acceptability 
of  performance  of  a  system.  For  MCS  testing,  three  types  of  governing  docu¬ 
ments  have  been  identified.  These  are  test  design  plans,  including  top  level 
and' detailed  test  design  requirements,  and  specific  test  procedures  for  the 
SUT;  performance  specifications  (A,  B,  and  C  level)  of  SUT;  and  specifications 
on  the  test  driver  system  ITIS  in  this  case. 

(1)  Test  Design  Documentation.  These  specifications  include  the 
TOP,  IEP,  TDP,  and  specific  Test  Procedures. 

(a)  TOP 

1_  The  US  Army  Test  and  Evaluation  Command  (TECOM)  TOP 
(Draft)  Computer  Software  Performance  Testing  (ref  11,  app  C)  lists  the  activ¬ 
ities  to  be  accomplished  prior  to  doing  tests;  these  include  preparing  for 
testing,  conducting  tests,  and  post-test  analysis  and  reporting.  This  proce¬ 
dure  does  not  tell  how  actions  are  to  be  done  but  rather  attempts  to  list  all 
actions  that  should  be  considered  when  planning  to  conduct  a  test.  It  in¬ 
cludes  a  number  of  applicable  checklists,  data  collection  sheets,  logs,  soft¬ 
ware  investigation  report  (SIR),  and  equipment  performance  report  (EPR)  forms 
to  be  used  when  planning  and  conducting  tests  and  analyzing  test  results. 

2_  Early  activities  include  definition  of  automated  support 
tool  requirements,  definition  of  data  reduction  analysis  (DRA)  requirements, 
definition  of  data  collection  requirements,  definition  of  scientific  and  engi¬ 
neering  data  processing  support  requirements,  data  reduction  and  analysis 
facilities  acquisition,  and  acquisition  of  automated  support  tools. 

_3  The  TOP  requires  implementation  of  a  test  configuration 
management  plan,  and  lists  the  items  to  be  included  in  the  plan.  Items  of 
interest  here  are  requirements  for  data  reduction  and  analysis  and  data 
processing  support,  test  data  handling  and  labeling,  and  test  related  informa¬ 
tion,  including  logs,  tapes,  and  checklists. 

Detailed  Test  Procedures  stress  the  need  to  determine 
the  quantity  and  precision  of  the  data  in  coordination  with  the  Data  Reduction 
and  Analysis  Team. 


5_  Data  reduction  includes  requirements  to  transform  test 
results  to  statistical  performance  measures  compatible  with  test  plan 
criteria,  isolate  performance  exception  measures  to  identify  SUT  performance 
anomalies,  and  the  ability  to  merge  data  from  different  test  categories. 
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6_  Data  analysis  requires  the  use  of  statistical  analysis 
techniques,  histograms,  regression  and  correlation  analysis,  and  tests  of 
significance.  It  further  requires  the  use  of  prepackaged  statistical  analysis 
routines  to  minimize  the  amount  of  user  generated  code. 

7  Data  presentation  requires  a  capability  to  present  data 
in  graphical  and  tabular  form.  It  also  requires  multi-function  plot  capabil¬ 
ity,  plot  scale  flexibility  and  appropriate  labeling,  and  presentation  of 
supporting  data. 

(b)  IEP 

1_  IEP  for  the  MCS  (ref  1,  app  C)  identifies  the  character¬ 
istics  of  the  primary  functional  objectives  of  the  MCS  and  the  associated  sub¬ 
objectives.  Development  testing  is  done  to  determine  the  capabilities  and 
limitations  of  the  MCS  as  they  relate  to  those  functional  objectives.  The  IEP 
identifies  the  major  functional  objective  of  the  force  level  and  MCS  to  be  the 
assurance  of  continuity  of  combat  operations.  To  achieve  this,  five  sub- 
objectives  must  be  met:  responsiveness,  survivability/security,  dependabil¬ 
ity,  flexibility,  and  interoperability. 

1_  Interoperability  identifies  the  critical  issue  as  the 
ability  of  the  MCS  to  Interoperate  with  all  systems  required  by  the  ABIC  with¬ 
out  causing  mutual  interference.  The  criteria  for  evaluation  is  that  the  MCS 
must  demonstrate  an  ability  to  establish  and  maintain  an  Information  exchange 
without  degrading  its  functional  performance  within  its  own  functional  area. 
The  criteria  for  timeliness,  accuracy  and  flexibility  established  for  maneuver 
unit  Information  processing  must  not  be  relaxed  when  interoperability  is 
achieved  with  other  Army  systems,  other  services'  systems,  and  other  NATO 
systems.  The  systems  with  which  MCS  must  interoperate  (per  this  IEP)  are: 

<a  Maneuver  Control  Functional  Area 

•  PLRS 

b_  Other  Functional  Areas 

«  ASAS 

•  TACFIRE 

«  CSS  C2 

•  SH0RAD-C2 

«  AN/TSQ-73 

°  PATRIOT 

•  SOTAS 

c_  Other  Services'  Systems 

°  WWMCCS 

«  TCO 

»  ITAWDS 

•  TACC  CAFMS 
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A.*. 
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ii  Allied  Systems 

°  WAVELL 
»  HEROS 
»  TACCDAS 

3  Data  requirements  and  sources  are  identified  as  follows: 
data  on  transmission  errors,  format  errors,  transmission  delays,  CPU  demands, 
message  volume,  rranslation/refomat  times,  and  response  times  for  information 
requests  for  MCS  will  be  collected  when  interoperating  with  each  system  for 
which  an  interface  has  been  specified  and  implemented  for  each  of  the  follow¬ 
ing  conditions: 


a  MCS  is  tasked  to  interoperate  with  other  systems 
and  has  no  maneuver  control  demands  placed  on  it. 

b  MCS  is  operating  under  a  most  likely  scenario  load 

of  tasks. 


e  MCS  is  operating  under  a  worst  case  loading. 

The  method  of  evaluation  will  be  to  analyze  test  data  to  determine  if  timeli¬ 
ness  criteria  for  the  interface  and  for  MCS  primary  functions  are 
satisfactorily  met. 

(c)  TDP 

The  TDP  (ref  13,  app  C)  serves  as  an  outline  for  the  re¬ 
quired  testing  to  be  done  during  Government  development  tests.  The  primary 
objectives  of  the  plan  are  minimization  of  testing  time  and  sample  size,  elim¬ 
ination  of  unnecessary  or  repetitive  testing,  and  maximum  usage  of  obtained 
results  in  evaluation  of  system  performance.  It  requires  that  mathematical 
and  statistical  procedures  be  used  on  test  data,  and  that  results  obtained 
with  these  techniques  be  used  in  evaluating  MCS  performance  regarding  categor¬ 
ies  in  the  IEP  and  in  determining  compliance  with  requirements  of  the  ROC 
document . 


1_  Concept  of  test  lists  three  questions  to  be  answered  re¬ 
garding  interoperability:  Can  the  MCS  interoperate  with  the  systems  as  out¬ 
lined  in  the  ROC?,  Can  MCS  interoperate  with  these  items  as  specified?,  Are 
the  specified  levels  of  interoperability  (automated,  human,  etc.)  currently  in 
existence  or  soon  to  be? 

3_  General  appro^h  includes  use  of  the  ITIS  to  satisfy 
testing  requirements.  It  lists  three  main  functions  of  the  ITIS,  including 
scenario  generation,  real-time  testing,  and  providing  post-test  data 
reduction/data  analysis  capability.  The  last  sentence  of  the  paragraph  re¬ 
quires  that  all  data  reduction  and  statistical  analysis  outlined  in  the  TDP 
should  be  provided  using  the  UCLA  BMDP-79. 

4  The  main  TDP  requirement  on  interoperability  testing 
states  test  objectives  to  be  to  ensure  that  the  MCS  and  each  of  its  interoper¬ 
ating  systems  (as  specified  in  the  ABIC)  have  implemented  an  Interface  and 
achieved  compatibility  and  can  interoperate  without  mutual  interference. 
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5^  It  requires  tests  to  be  run  with  the  MCS  interoperating 
with  one  or  more  systems  in  each  of  the  following  conditions: 


it. 

functions, 
tasks . 


£  MCS  having  no  maneuver  control  demands  placed  on 
is  MCS  minimally  tasked  with  maneuver  control 
c_  MCS  operating  under  a  most  likely  scenario  load  of 
d  MCS  operating  under  a  worst  case  loading. 


It  should  be  noted  that  the  second  item  is  an  additional  requirement  over 
those  stated  in  the  IEP. 

6.  The  paragraph  on  data  requirements/data  presentation 
states  that  the  test  report  shall  provide: 


message  type. 


£  Number  and  type  of  transmission  errors  for  each 


type. 


_b  Number  and  type  of  format  errors  for  each  message 


£  Transmission  delays  for  each  message  type  for  each 
system  for  each  set  of  test  conditions. 

d_  The  volume  and  frequency  of  messages  by  message 
type  from  each  node  of  each  system  and  to  each  node  of  each  system. 

£  The  response  times  of  each  specific  type  of  demand 
made  on  MCS  and  by  MCS  for  each  set  of  test  conditions. 

f_  A  description  of  the  integrated  prioritization  of 
demands  (both  internally  and  externally  generated)  for  MCS  to  service  (priori¬ 
tization  specified  by  the  proponent)  and  the  results  of  tests  run  to  demon¬ 
strate  that  the  prioritization  has  been  implemented. 


7_  The  following  paragraph  does  not  address  interoperabili¬ 
ty  requirements  directly,  but  is  mentioned  here  as  additional  information. 
Paragraph  3.2.1. 1,  System  Performance,  sub-issue  k,  communications  compatibil¬ 
ity  and  sub-issue  1,  multiple  peripheral  devices,  present  some  additional 
requirements  in  their  sections  on  data  requirements/data  presentations.  For 
example,  item  k,  (3)  £  states  that  error  statistics  for  messages  sent/received 
across  the  interface  both  with  and  without  error  control  encoding  are 
required. 


(d)  IEP/TDP  for  the  TCT  and  TCS.  This  document  (ref  12,  app  C) 
was  reviewed  since  the  MCS  is  actually  composed  of  combinations  of  TCTs  and 
TCSs,  In  section  II,  the  IEP  portion,  section  D,  data  requirements,  the 
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discussion  is  very  general,  and  does  not  specifically  address  compatibility  or 
interoperability  testing.  Likewise,  in  section  III,  the  TDP,  the  subjects 
interoperability,  data  requirements/data  presentation  are  nowhere  addressed. 
Therefore,  this  document  does  not  identify  any  user  needs  or  requirements  for 
automation  of  post-test  data  analysis. 

(e)  SUT  Specific  Test  Procedures 

1  USAEPG  Test  Procedure  2.5.5,  Message  Prioritization 
(ref  16,  app  C)  is  considered  here  as  an  example  of  the  detailed  document  that 
is  required  to  run  each  test.  This  Joeument  contains  130  pages,  of  which  108 
pages  are  detailed  step-by-step  instructions  for  the  test  operator.  However, 
there  are  10  pages  on  scenario  requirements  which  specify  details  on  the  time 
of  a  message,  its  source,  destination,  and  precedence.  This  information  is 
used  in  building  the  message  scenario  tape. 

2  Testing  and  post-test  data  reduction  and  analysis  are 
separate  functions;  the  main  item  of  interest  is  the  generat  a  of  the  test 
data  base  (tape  or  other)  and  the  availability  of  that  data  base  for  the  data 
reduction  and  analysis  function.  Therefore,  the  specific  test  procedure 
affects  automation  of  interoperability  test  data  analysis  to  the  extent  that 
it  allows  quick  availability  of  a  test  data  base,  and  that  it  influences  the 
selection  of  the  format  and  medium  of  the  test  data  base. 

(2)  MCS  Performance  Specification 

(a)  In  general,  one  would  expect  that  there  would  be  a  perform¬ 
ance  specification  on  the  MCS  with  detailed  performance  requirements  to  be  met 
by  the  item  as  delivered  by  the  supplier.  For  example,  the  PLRS  Final  Report 
references  a  NAVCON  document  that  appears  to  be  the  PLRS  performance  specifi¬ 
cation.  To  date,  no  single  document  has  been  identified  as  being  the  MCS 
System  Specification.  However,  the  first  part  of  the  TDP  (ref  13,  app  C)  in¬ 
cludes  a  paragraph  1.3,  Proposed  System  Requirements,  taken  from  a  draft  ROC, 
which  appears  to  be  the  actual  requirements  to  which  the  MCS  was  designed  and 
built . 


(b)  The  main  requirements  of  interest  are  the  following: 

1_  Exchange  information  between  all  echelons  from  battalion 
to  corps.  The  priorities  of  the  information  areas  are  mission,  friendly  situ¬ 
ation,  operational  environment,  and  enemy  situation. 

2_  Provide  for  continuity  of  operations  to  insure  that 
critical  information  will  be  available  at  command  posts  when  devices  servicing 
the  command  posts  and/or  communications  supporting  those  devices  fail  or  are 
being  moved. 


3  Be  capable  of  utilizing  existing  and  planned  Army  tacti¬ 
cal  communications  as  of  1985  and  must  not  create  a  unique  electromagnetic 
signature  when  introduced. 
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4  Provide : 


£  A  capability  to  edit,  compose,  and  validate 
messages.  Prompts  to  assist  message  composition,  to  include  error  prompts, 
will  also  be  provided. 

b_  When  loads  exceed  peak,  discontinue  message  proces¬ 
sing  and  the  flow  of  information  items  in  inverse  order  of  precedence.  The 
lower  priority  items  must  be  retained  in  storage  at  the  originating  device  and 
be  transmitted  when  the  load  decreases,  unless  purged  by  the  system.  The 
local  device  must  be  notified  that  the  data  has  not  been  transmitted. 

5_  Be  capable  of  interoperating  with  systems  as  identified 

in  the  ABIC. 

<S  Be  capable  of  simultaneous  reception  and  transmission  of 
information  without  interference  with  message  preparation. 

(3)  Test  Driver  (ITIS)  Specifications.  The  performance  requirements 
of  the  ITIS,  the  system  currently  being  used  for  MCS  interoperability  testing, 
are  contained  in  a  number  of  specifications.  The  area  of  post-test  data  anal¬ 
ysis  is  covered  in  Computer  Program  Configuration  Item  (CPCI)  Specification, 
Communications  Applications  Real-Time  Remotely  Operable  Test  System  (CARROTS) 
for  Tactical  Automated  Test,  Evaluation,  and  Reporting  System  (TATERS) 

(ref  14,  app  C)  and  in  CPCI  Specification,  Post-Teat  Evaluation  Analysis  and 
Reporting  System  (PEARS)  for  TATERS  (ref  15,  app  C). 

(a)  CARROTS  Specification 

1_  This  specification  is  mainly  concerned  with  the  real¬ 
time  test  driver,  which  includes  generation  of  the  test  log  tape.  The  infor¬ 
mation  on  tne  log  tape  is  the  input  data  to  the  post-test  analysis  system. 

2_  CARROTS  Log  Tape  Interface  presents  a  figure  showing  the 
CARROTS  log  tape  format,  and  a  table  showing  the  CARROTS  log  tape  data.  The 
data  item  shows  the  log  tape  records  contents,  and  indicates  three  types  of 
records:  header,  data,  and  trailer. 

3_  The  CARROTS  specification  is  dated  18  July  1979  and 
presents  the  state  of  the  log  tape  formats  at  that  time.  The  Bell  Technical 
Operations  Letter,  subject:  Sample  ITIS  Test  Data,  dated  19  November  1982 
(ref  4,  app  C)  gives  more  detailed  information  on  the  log  tape  formats.  These 
include  a  teat  identification  record  and  a  test  description  record  at  the 
start  of  the  tape,  eleven  different  types  of  message  and  message  related 
records,  and  an  operator  log  record  and  a  trailer  record  at  the  end  of  the 
recorded  data.  This  letter  also  includes  much  information  on  record  content 
although  it  is  not  complete.  The  record  identifications  and  associated 
channels  are  shown  in  table  II. 
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TABLE  II.  MESSAGE  LOG  TAPE  RECORD  TYPES  AND  CHANNELS 


Channel 

Record  Type 

10 

Test  Identification 

11 

Test  Description 

30 

Input  Transmit  Message 

31 

Transmit  Message  or  Service  Message 

32 

Transmit  Acknowledge  Message 

42 

Decoded  Transmit  Acknowledge 

33 

Received  Message  and  Service  Messages 

43 

Decoded  Received  Text  Message 

34 

Received  Acknowledge  Match 

44 

Decoded  Received  Acknowledge  Match 

35 

Received  Acknowledge  No  Matches 

45 

Decoded  Received  Acknowledge  No  Match 

46 

Decoded  Received  Service  Message 

40 

Operator  Log  Records 

51 

Trailer  Record 

(b)  PEARG  CPCI  Specification 

1_  This  speci f icaticn  (ref  15,  app  C)  established  the 
functional  requirements  for  performance,  design,  test,  and  acceptance  of  the 
post-test  evaluation,  analysis  and  reporting  system.  This  system  is  required 
to  generate  reports  containing  information  necessary  for  analysing  and  evalu¬ 
ating  MCS  performance  in  response  to  scenarios  on  the  MST.  This  specification 
required  the  PEARS  to  read  and  analyze  both  MST  and  MLT .  The  following  re¬ 
ports  were  required  to  be  generated  by  the  PEARS:  message  log  presentation, 
time/event  synopsis,  test  configuration  report,  exceptions  report,  graphics/ 
data  messages  reports,  and  statistical  analysis  report. 

2_  The  PEARS  specification  is  dated  2  October  1979  and 
reflects  the  state  of  the  post-test  analysis  system  as  of  that  date.  The  PTAS 
User's  Manual  (ref  5,  app  C)  gives  more  up-to-date  information  on  the  current 
capabilities  of  the  post-test  analysis  system  on  ITIS.  These  are  discussed 
more  fully  in  paragraphs  2.2a  and  2.3b(2). 

b.  Current  Testing  and  Analysis.  The  current  MCS  testing  is  done  using 
various  configurations  of  TCSs  and  TCTs  and  a  number  of  different  scenarios. 
The  post-test  analysis  of  the  data,  as  it  is  being  done  at  present,  uses  some 
of  the  available  features  of  the  post-test  analysis  system-  The  testing  and 
analysis  are  discussed  in  the  following  sections. 

(1)  MCS  Interoperability  Testing 

(a)  Paragraph  3.1  of  the  MCS  TDP  (ref  13,  app  C)  specifies  two 
MCS  configurations  to  be  used  during  interoperability  testing.  Subtest  I  uses 
one  TCS  and  one  TCT;  Subtest  II  uses  three  TCSs  and  three  TCTs.  Subtest  III 
is  the  same  configuration  as  that  of  subtest  II,  but  requires  testing  in  a 
more  representative  tactical  field  environment  using  existing  communications. 
In  addition,  the  TDP  states  that  use  of  the  ITIS  whenever  necessary  to  satisfy 
testing  requirements  is  implicit  in  the  test  plan. 
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(b)  The  message  scenario  tapes  were  used  to  conduct  a  number  of 
different  types  of  tests,  including  peak  load,  message  reception,  and  message 
prioritization  tests.  These  tapes  include  information  on  ten  types  of  tests; 
the  test  type  and  number  of  tests  for  that  type  are  shown  in  table  III. 

TABLE  III.  MCS  SCENARIO  INFORMATION 


Test  Type 

Number  of 
Tests  Run 

Peak  Load 

59 

Message  Reception 

47 

Message  Prioritization 

47 

Remote  User  Impact 

31 

Message  Prep  Interface 

26 

Miscellaneous  (Dry  Run  or 

26 

MCS  System  Checkout) 

DBMS 

8 

Initialization 

7 

Networks 

3 

Distribution  Lists 

_ 2 

Total  Number  of  Tests  Run 

256  | 

(c)  Some  information  on  the  content  of  the  message  log  tape  was 
presented  in  paragraph  2.3a(3)(a).  As  indicated  there,  the  test  data  include 
up  to  eleven  different  types  of  message  and  message  related  records.  The  con¬ 
tents  of  each  of  these  record  types  is  given  in  the  Bell  Technical  Operations 
letter  (ref  4,  app  C). 

(d)  If  there  are  an  average  of  300  messages  per  run,  then  the 
total  number  of  messages  to  be  processed  and  analyzed  is  on  the  order  of 
75,000.  This  quantity  of  tests  and  messages  is  such  that  a  complete  and 
thorough  data  reduction  and  analysis  requires  further  automation  of  the  PTAS 
process. 


(2)  Test  Data  Analysis 

(a)  Two  primary  pieces  of  information  available  to  the  analyst, 
when  assessing  SUT  performance,  are  the  teat  procedure  and  the  message  log 
tape.  Also,  there  would  be  printed  outputs  from  the  TCS  and  test  conductor 
log.  The  tool  available  to  do  MLT  analysis  is  the  PTAS.  The  analyst  must  be 
familiar  with  the  test  objectives,  SUT  and  test  driver  configurations,  and  ex¬ 
pected  outputs  to  do  a  proper  evaluation  of  SUT  performance. 

(b)  The  PTAS  enables  extraction  of  selected  information  from 
the  MLT  and  presentation  of  that  data  in  a  suitable  format,  e.g.,  dump, 
tabular,  plot,  or  histogram.  Use  of  the  PTAS  is  described  in  the  PTAS  User's 
Manual  (ref  5,  app  C).  In  addition,  the  PTAS  allows  creation  of  a  file  in  the 
proper  format  so  that  the  Biomedical  Data  Package  (BMDP)  software  package  can 
be  used  to  do  statistical  analysis  and  plotting  the  data. 
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(c)  As  already  discussed  in  paragraph  2.2a,  the  analysis  of 
data  using  PTAS  is  controlled  by  the  specification  of  inputs  for  up  to  five 
control  tables:  format,  primary  variable,  dump,  table,  and  plot.  The  PTAS 
User's  Manual  shows  sample  runs  to  build  the  five  control  tables;  in  the 
samples  shown,  the  user/operator  must  enter  information  on  the  number  of  lines 
shown  as  follows: 


Format  Control  Table  17 
Primary  Variable  Control  Table  11 
Dump  Control  Table  10 
Table  Control  Table  10 
Plot  Control  Table  15 


(d)  These  are  approximate  minimums  required,  and  for  dumps/ 
tables/plots  having  more  variables,  more  entries  are  required.  Thus,  the 
current  PTAS  software  has  two  significant  features:  it  is  very  flexible  as  to 
selection  of  inputs  and  outputs;  this  very  flexibility  means  that  the  current 
PTAS  system  is  labor  intensive,  requiring  35  to  40  or  more  entries  by  a  user 
in  order  to  get  a  product. 

(e)  The  first  sentence  of  paragraph  2.5  of  the  PTAS  User's 
Manual  states  that  "the  PTAS  provides  an  automated  general-purpose  data 
reduction  capability  for  the  ITIS."  Thus  "automation"  of  post-test  data 
analysis  would  seem  to  be  open  to  some  interpretation,  and  perhaps  a  strict 
definition  of  "automated"  interoperability  post-test  data  analysis  is  needed. 
In  ITIS  testing  and  data  reduction  and  analysis,  automated  means  the  genera¬ 
tion  of  a  predetermined  set  of  analysis  products  with  a  minimum  number  of 
operator  inputs  (e.g.,  5  or  less).  In  view  of  the  many  operator  inputs 
required  by  the  current  PTAS,  it  does  not  appear  to  be  a  fully  automated 
system,  but  rather  a  semi-automated  system  with  built  in  flexibility  to  allow 
analysts  to  change  Inputs  and  outputs  as  the  analysis  progresses. 

(f)  The  above  information  is  Indicative  of  the  PTAS  system  as 
it  was  designed  and  coded.  However,  discussion  with  personnel  doing  the 
analysis  of  the  current  MCS  test  data  indicates  that  most  of  the  features 
discussed  are  not  being  used;  in  fact,  the  current  PTAS  usage  is  limited  to 
production  of  log  tape  dumps  with  the  following  information:  input  transmit 
message,  decoded  transmit  acknowledge,  decoded  received  text  message,  decoded 
received  acknowledge  match,  decoded  received  acknowledge  unmatched,  decoded 
received  service  message,  test  identification,  ard  test  description.  Of  this 
tape  dump,  some  comparisons  among  fields  are  being  done  to  produce  tabular 
data  by  hand;  no  computer  generated  plots  are  being  done.  Thus,  there  is  only 
small  usage  of  even  the  present  PTAS  capabilities.  It  should  be  stated  here 
that  these  facts  are  to  some  extent  influenced  by  the  particular  performance 
of  the  SUT  in  test  (MCS)  and  that  for  other  SUTs  the  amount  of  usage  of  the 
PTAS  could  be  more  complex  and  complete.  A  well  designed  and  automated  PTAS 
will  function  and  generate  useful  products  even  If  the  SUT  is  working  poorly, 
hanging  up  or  aborting  the  test  for  other  reasons. 
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2.4.  AUTOMATION  OF  POST-TEST  DATA  ANALYSIS 


The  second  objective  of  the  AIOPTDA  task  is  the  proposal  of  an  appro¬ 
priate  approach  for  the  automation  of  interoperability  post-test  data  analysis 
for  the  MCS.  The  approach  would  provide  an  effective  method  for  reduction  and 
analysis  of  typical  test  data  from  a  large  data  base  generated  during  inter¬ 
operability  testing.  Two  approaches  are  discussed  in  this  section.  The  first 
is  an  extension  of  the  present  PTAS  to  make  it  more  fully  automated.  The 
second  is  the  use  of  embedded  query  calls  in  a  commercially  available  data 
base  management  system. 

a.  Extension  of  Current  PTAS.  The  method  of  operation  of  the  PTAS  is 
outlined  in  the  PTAS  User's  Manual  (ref  5,  app  C),  and  has  already  been 
discussed  in  paragraphs  2.2a  and  2.3b(2).  The  essential  elements  of  operating 
PTAS  are  shown  in  figure  3;  operator  inputs  are  done  in  an  interactive  mode, 
using  the  PTAS  to  extract  information  from  the  message  log  tape.  The  PTAS 
outputs  are  dumps,  tables  and  plots,  or  save  files.  The  information  in  save 
files  can  be  used  as  inputs  to  the  BMDP  package  for  further  statistical 
analysis  and  plotting. 

(1)  Use  of  Runstream 

(a)  The  proposal  to  use  an  extension  of  the  current  PTAS  uses  a 
predefined  runstream  to  perform  automatically  the  operations  and  inputs  that 
would  be  done  by  an  operator  sitting  at  a  terminal.  The  runstream  is  a  file 
whose  contents  emulate  the  actions  an  operator  would  take  when  running  the 
PTAS  software,  or  more  generally,  when  generating  a  product  (dump,  table, 
plot,  save  file)  using  the  MLT  data  as  Input.  The  runstream  exists  as  a  stand 
alone  program,  and  when  initiated,  will  require  a  minimum  number  of  inputs, 
e.g.,  Input  message  log  tape  number  or  the  tape  drive  on  which  it  is  mounted. 
Each  runstream  would  have  a  predefined  output,  either  a  single  dump,  table  or 
plot,  or  some  combination  of  these;  else  It  would  create  a  save  file  and  auto¬ 
matically  call  the  BMDP  software  for  further  processing.  Actual  implementa¬ 
tion  of  such  a  scheme  is  hardware  and  operating  system  dependent,  but  has 
already  been  done  on  two  systems  to  this  writer’s  knowledge  (a  Varian  mini  and 
Univac  large  frame)  and  has  probably  been  done  on  others. 

(b)  The  main  features  of  the  predefined  runstream  are  inclusion 
of  all  calls  to  stand  alone  programs,  inclusion  of  all  data  inputs  required 
(pseudo  operator  inputs),  and  a  fixed  (predetermined)  output  or  set  of  out¬ 
puts.  This  method  of  automating  post-test  data  analysis  uses  the  test  data  in 
the  form  as  it  is  currently  created:  either  in  a  disk  file  or  on  magnetic 
cape.  Further,  present  PTAS  software  is  used  as  it  exists  at  present  or  with 
only  minor  modifications  to  the  main  programs. 

(2)  Definition  of  Outputs 

(a)  Automation  of  data  processing  has  an  immediate  implication 
for  those  who  would  like  to  retain  great  flexibility  when  doing  data  analysis: 
some  of  the  flexibility  must  be  surrendered.  The  very  nature  of  the  automa¬ 
ting  process  requires  hard  definitions  of  the  output  products  desired. 
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(b)  A  large  number  of  message  log  tapes  were  generated  during 
the  DT  testing.  One  usage  of  PTAS  was  to  generate  a  standard  dump  of  selec¬ 
ted  data  from  those  tapes  for  use  by  the  analysts.  Thus,  there  has  been  an 
evolution  to  one  type  of  standardized  output  already.  It  Is  this  type  of 
standardization  or  output  product  definition  that  must  be  done  for  further 
implementation  of  automation  of  the  MCS  test  data  processing.  Once  a  desired 
output  product  has  been  defined,  it  is  a  straightforward  process  to  define  the 
runstream  required  to  produce  that  product. 

(3)  Use  of  BMDP.  The  use  of  a  runstream  to  run  the  PTAS  software 
can  include  the  automatic  creation  of  a  save  file  for  BMDP  use.  The  runstream 
would  include  the  call  to  the  required  BMDP  software  package  to  do  the  statis¬ 
tical  analysis  or  plotting  desired.  (See  the  right  side  of  figure  3.)  The 
important  point  to  note  as  before  is  the  need  to  have  predefined  the  output 
product  or  products  that  are  required. 

(4)  Implementation  Resources  Required 

(a)  There  are  five  areas  of  consideration  when  implementing  the 
above  scheme  for  automation  of  test  data  analysis: 

1_  Data  base  definition 

2_  PTAS  software  definit'on 

3_  Runstream  implementation 

4_  Output  product  definition 

5^  Demonstration  of  the  scheme 

(b)  The  first  item,  data  base  definition,  would  require  little 
or  no  modificaion  to  the  present  data  base.  At  most,  it  might  be  desirable  to 
be  able  to  operate  directly  off  of  the  log  data  on  disk  rather  than  after  it 
has  been  dumped  to  tape.  At  any  rate,  a  minimal  effort  is  required  in  this 
area . 


(c)  The  second  item,  the  PTAS  software,  likewise  should  require 
only  minor  modifications  to  allow  operation  in  this  mode.  The  inclusion  of 
the  BMDP  capability  might  add  some  effort  required  to  this  task,  but  should  be 
of  a  similar  magnitude  to  the  PTAS  mods.  The  modifications  required  are  those 
related  to  the  transition  from  an  interactive  terminal  with  operator  inputs  to 
automated  inputs  on  a  pregenerated  parameter  file.  The  main  point  is  to  keep 
the  PTAS  and  BMDP  software  modular  and  general  so  that  the  same  modules  are 
used  in  each  runstream  or  command  file  with  only  the  input  parameter  files 
being  different. 

(d)  The  third  item,  the  runstream  implementation,  as  already 
pointed  out,  is  hardware  and  operating  system  dependent.  In  the  usual  situa¬ 
tion,  a  small  effort  by  the  system  programmer  should  be  sufficient  to  allow 
implementation  of  this  mode  of  operations,  if  it  does  not  already  exist  on  the 
system. 
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(e)  The  fourth  item,  output  product  definition,  will  be  the 
most  difficult  one  to  accomplish.  It  will  require  inputs  and  cooperation  of 
all  those  responsible  for  determining  how  well  the  SUT  meets  its  performance 
requirements,  including  the  test  conductor,  analysts,  and  report  writers.  It 
is  difficult  to  estimate  the  extent  of  effort  required  to  accomplish  this;  in 
part,  it  would  at  first  be  an  on-going  effort  until  satisfactory  products  had 
been  identified. 

(f)  The  fifth  item,  demonstration  of  the  scheme,  is  done  as 
soon  as  the  first  three  items  have  been  accomplished  and  at  least  one  end 
product  has  been  identified.  This  demonstration  serves  as  proof  of  the 
feasibility  of  automating  the  test  data  analysis  in  this  manner. 

b.  Alternate  DBMS.  The  approach  to  automation  presented  in  paragraph 
2.4a  is  based  on  the  present  ITIS  configuration  where  there  are  two  data 
bases,  one  for  message  scenario  tape  generation  and  the  second  containing  the 
log  tape  data.  Here  we  consider  an  alternate  scheme  where  the  two  data  bases 
are  replaced  by  one  data  base,  probably  a  commercially  available  DBMS. 

(1)  DBMS  Features 

(a)  The  current  generation  of  DBMSs  include  a  number  of  fea¬ 
tures  of  interest  for  this  study.  These  features  include  data  base  structure, 
inclusion  of  query  and  embedded  query  language,  and  inclusion  of  a  data 
analysis/grsphics  package. 

(b)  The  three  models  used  In  the  representation  of  data  are 
relational,  network,  and  hierarchical.  A  comparison  of  these  models  is  not 
appropriate  here;  a  discussion  is  provided  on  pages  168-170  of  Principles  of 
Data  B-.ae  Systems  (ref  18,  app  C).  One  point  of  the  reference  is  that  for 
smaller  Ar  tn  braes,  on  the  order  of  thousands  or  tens  of  thousands  of  records, 
ease  of  us*',  is  u  prime  consideration  and,  in  this  case,  the  relational  model 
is  the  best.  Writing  applications  programs  and  phrasing  queries  is  easier  and 
can  be  dune  by  .  rsons  with  lesser  programming  skills  In  the  relational  model. 
For  larger  data  bases,  where  efficiency  of  implementation  is  an  important 
factor,  the  network  and  hierarchical  models  are  better.  On  the  basis  of  the 
quantity  of  i  .  lges  presented  in  paragraph  2.3b(l),  the  current  ITIS  data 
base  falls  in  ■  the  smaller  category  with  thousands  of  records  per  MCS  test. 
However,  this  sizing  could  grow  if  other  systems  are  tested  (e.g.,  TACFIRE) 
and  also  If  test  data  are  allowed  to  accumulate  on  the  data  base. 

(c)  Data  definition  and  data  manipulation  are  done  by  a  data 
manipulation  language,  commonly  called  a  query  language.  In  a  relational  data 
base,  the  query  language  is  non-procedural;  the  user  specifies  what  is  to  be 
done,  but  not  how  it  is  to  be  done.  Such  a  language  allows  the  following 
operations  on  the  data  base: 

1  Data  Definition 

Create  tables 
Destroy  tables 
Modify  storage  structures 
Add  or  delete  secondary  structures 
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2_  Data  Manipulation 
a_  Print  tables 

b  Retrieve  data  from  tables  based  upon  conditions 
£  Append  rows  to  tables 

d  Delete  rows  from  tables 

e_  Change  data  in  tables 

(d)  For  post-test  data  analysis,  the  operation  of  most  interest 
would  be  the  second  one  under  data  manipulation,  the  retrieval  of  data  from 
tables  based  on  some  set  of  conditions. 

(e)  The  third  feature  of  current  DBMSs  is  the  inclusion  of  a 
data  analysis/graphics  package.  Selection  of  a  DBMS  with  such  a  feature  would 
allow  a  direct  tie-in  of  the  query  language  with  the  generation  of  graphics 
data.  For  example,  the  Interactive  Graphics  and  Retrieval  System  (INGRES) 
graphics  package  allows  several  graph  types  including  bar  charts,  pie  charts, 
scatter  plots,  and  general  line  graphs;  and  other  features,  Including  head¬ 
ings,  legends,  cross-hatch  patterns,  dot  types,  line  types,  type  faces, 
titles,  linear  regression  for  scatter  plots,  scales  for  quantities  including  a 
logarithmic  scale  and  tick  marks  for  various  scales.  This  package  is  support¬ 
ed  on  a  number  of  output  devices,  including  DEC  VT125,  Tektronix  40XX,  AED 
412/7C7,  and  RAMTEK  62XX  terminals;  Zeta,  Calccnnp,  and  Hewlett-Packard  plot¬ 
ters;  and  the  Dicomed  film  recorder. 

(2)  Post-Test  Data  Analysis  on  a  DBMS 

(a)  How  do  we  do  post-test  data  analysis  when  the  data  are 
sitting  in  a  general  data  base?  That  Is  the  question  to  be  addressed  here. 

As  already  noted,  current  generation  DBMSs  will  have  some  sort  of  query 
language  for  retrieval  of  selected  data;  In  addition,  we  shall  require  that 
the  DBMS  will  have  an  embedded  query  capability.  This  means  that  the  query 
commands  can  be  embedded  in  an  applications  or  host  program  written  in  a  high 
order  language  (HOL)  such  as  Fortran,  Cobol,  Basic,  Pascal,  or  C.  The  key  to 
this  capability  is  the  availability  of  a  preprocessor  or  precompiler  which 
replaces  the  query  language  calls  with  normal  HOL  calls  after  which  the  code 
can  be  compiled  in  the  normal  manner. 

(b)  The  philosophy  here  is  to  avoid  rewriting  the  PTAS  data 
selection  software  In  terms  cf  the  query  language;  instead,  we  want  to  write 
an  applications  program  for  each  analysis  product  identified.  After  all,  the 
premise  above  is  that  query  is  easy  to  use,  especially  with  a  relational  data 
base;  in  effect,  we  are  replacing  the  PTAS  data  selection  software  with  the 
use  of  the  query  language. 

(c)  As  to  the  resources  required  to  implement  this  approach, 
highly  sophisticated  programmers  should  not  be  required  for  Implementation, 
and  the  total  cost  would  be  directly  proportional  to  the  number  and  complexity 
of  products  requested. 
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APPENDIX  A.  METHODOLOGY  INVESTIGATION  PROPOSAL 


1.  TITLE.  Automation  of  Interoperability  Test  Data  Analysis. 

2.  CATEGORY .  Interoperability  Tests. 

3.  INSTALLATION.  US  Army  Electronic  Proving  Ground,  Fort  Huachuca,  Arizona 
85613. 

4.  PRINCIPAL  INVESTIGATOR.  Max  Sanders,  Software  and  Automation  Branch, 
STEEP-MT-DA,  AUTOVON  879-6957,  6058,  6870. 

5.  STATEMENT  OF  THE  PROBLEM.  A  rapidly  increasing  number  of  automated  C-E 
systems  are  being  developed  for  use  by  the  field  army.  The  comprehensive 
testing  of  the  interoperability  of  these  complex  systems  is  critical  to  their 
performance  evaluation.  Such  testing  will  generate  large  volumes  of  test 
data.  Manual  analysis  is  incomplete,  time  impractical,  and  excessively  costly 
to  conduct.  Automated  data  analysis  test  methodologies  must  be  developed 
which  will  optimize  the  use  of  automated  data  analysis  technology  to  provide  a 
complete,  timely,  and  economically  practical  evaluation  capability. 

6.  BACKGROUND .  The  Army  and  DOD  have  developed  and  are  continuing  to  develop 
a  number  of  major  automated  C3I  Systems.  These  include  TACFIRE,  MCS,  TCAC, 
TCS/TCT,  AN/TSQ-73,  REMBASS,  SOTAS,  and  ASAS.  These  systems  are  designed  to 
interface  with  a  large  number  of  interactive  systems  and  to  operate  in  a 
highly  interactive  environment.  The  critical  element  in  the  success  or  fail¬ 
ure  of  these  C3I  Systems  will  be  their  interoperability  and  performance  under 
load  in  a  highly  interactive  tactical  environment.  The  nature  of  Interopera¬ 
bility  testing  results  In  the  generation  of  large  volumes  of  data.  A  need 
exists  to  develop  an  automated  data  analysis  capability.  The  enormous  cost  of 
manual  analysis,  both  in  dollars  and  test  period,  is  prohibitive.  Additional¬ 
ly,  the  added  probability  of  multiple  errors  in  manual  analysis  makes  such 
analyses  questionable.  The  Army  is  developing  major  systems  with  a  critical 
role  in  the  combat  effectiveness  of  the  1982  and  beyond  field.  The  Inability 
to  automatically  analyze  and  evaluate  the  data  generated  in  interoperability 
testing  is  a  risk  that  the  Array  cannot  afford  to  take. 

7.  GOAL.  This  investigation  will  develop  comprehensive  and  cost  effective 
test  methods  for  reducing  and  analyzing  the  large  volume  of  test  data  genera¬ 
ted  during  interoperability  testing  of  automated  field  army  C-E  systems. 

8.  DESCRIPTION  OF  INVESTIGATION 


a.  This  investigation  will  provide  the  foundation  for  the  analysis  of 
interoperability  test  data  for  the  Army.  The  investigation  will  identify 
critical  data  analysis  deficiencies  and  develop  solutions  to  resolve  them. 

The  data  that  needs  to  be  collected  will  be  identified,  and  the  manner  in 
which  it  can  be  correlated  to  the  input  will  be  determined.  This  is  to  in¬ 
clude  providing  traceability  from  output  data  through  the  input  and  back  to 
the  specification.  A  generalized  solution  to  the  complex  problem  of  analyzing 
these  large  amounts  of  data  will  be  sought  such  that  a  cost  effective  but 
empirically  validated  interoperability  evaluation  results. 
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b.  USAEPG  will  conduct  the  overall  investigation  in  two  phases.  Phase 
I  will  cover  a  basic  problem  analysis  and  solution  study  and  Phase  II,  a 
measurement  system  design  development. 

(1)  Phase  I 

(a)  Current  Technology  Search.  A  review  of  existing  reportage 
covering  data  reduction  and  analysis  to  include  requirements  documents  for  the 
1982-1992  time  frame  automated  tactical  C-E  systems. 

(b)  Problem  Definition.  The  separation  and  determination  of 
the  key  data  analysis  deficiencies  which  prevent  the  effective  assessment  of 
the  data  collected.  This  is  to  include  the  determination  of  the  data  which 
must  be  collected. 

(c)  Solution  Development.  The  identification  and  evaluation 
of  potential  solutions  and  the  detailing  of  the  particular  set  of  data 
collection,  reduction,  and  analysis  techniques  required.  Economic  as  well  as 
technological  problems  will  be  addressed. 

(2)  Phase  II 

(a)  Measurement  Requirements.  The  development  of  the  manner  in 
which  the  output  can  be  correlated  with  the  input  and  specifications.  This 
will  provide  traceability  from  output  data  through  the  input  and  back  to 
specification. 


(b)  Measurement  System  Design.  Developing  the  detailed  design 
of  the  overall  data  analysis  system.  The  system  will  incorporate  current 
USAEPG  capabilities  to  the  extent  possible.  Selection  and  purchase  of  analy¬ 
sis  capabilities  will  also  be  considered.  Methodology  for  validating  the 
chosen  approach  will  be  included. 


c.  USAEPG  will  conduct  the  investigation  as  follows: 


MILESTONE /PHASE 


Current  Technology  Search 
Problem  Definition 
Solution  Development 
Measurement  Requirements 
Measurement  System  Design 
Report 


SCHEDULE 

FY  82  (Qtrs)  FY  03  (Qtrs) 

1  2  3  4  1  2  3  4 

x  x 

X  X 

X 

X 

XXX 

X 


l 


d.  The  investigation  will  result  in  establishing  procedures  for  analysis 
of  interoperability  testing  data.  The  procedures  will  provide  utilization  of 
automatic  data  analysis  techniques  in  Interoperability  evaluations. 

e.  Environmental  Impact  Statement.  Execution  of  this  task  will  not  have 
an  adverse  impact  on  the  quality  of  the  environment. 
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f.  Health  Hazard  Statement.  Execution  of  this  task  will  not  involve 
health  hazards  to  personnel. 

9.  PROGRESS.  New  investigation. 

10.  JUSTIFICATION 


a.  Mission  and  Impact  Statement 

(1)  Association  with  Mission.  USAEPGs  primary  mission  is  to  conduct 
development  testing  of  C-E  equipment  and  systems.  In  support  of  this  mission, 
USAEPG  has  developed  extensive  experience  in  compatibility,  vulnerability, 
electronic  warfare,  and  intelligence  testing,  automated  system  (software) 
testing,  and  more  recently  pioneered  efforts  in  interoperability  testing. 
Automated  data  analysis  represents  the  logical  extension  and  syntheses  of 
interoperability  testing.  Development  of  automatic  analysis  methodology  will 
allow  an  effective  utilization  of  testing  data.  Without  this  methodology,  the 
analysis  must  be  accomplished  by  manual  means. 

(2)  Present  Capability,  Limitations,  Improvement,  and  Impact  on  Test 
if  Not  Approved.  USAEPG  has  a  number  of  key  methodology  instrumentation  and 
test  facility  elements  required  for  a  successful  interoperability  test  capa¬ 
bility.  These  elements,  however,  do  not  have  an  automatic  test  data  analysis. 
To  provide  a  cost  effective  data  analysis  capability,  the  current  manual  capa¬ 
bilities  at  USAEPG  must  be  upgraded  to  automated  data  analysis  methodology. 

If  this  task  is  not  approved,  USAEPG  will  have  to  reduce  data  by  manual  means 
at  an  enormous  cost  in  manhours  and  extended  test  periods. 

b.  Dollar  Savings.  No  direct  cost  savings  can  be  computed  at  this  time. 
However,  experience  has  demonstrated  that  the  only  alternative  to  the  proposed 
automated  data  analysis  approach  is  manual  analysis.  It  is  conservatively 
estimated  that  the  latter  could  easily  consume  one-third  of  USAEPGs  fiscal 
resources  for  the  entire  year  for  one  major  C3I  system  interoperability  test. 

c.  The  following  major  field  array  automated  systems  are  currently  under 
development  and  are  programmed  for  testing  during  the  1982  to  1992  time  frame. 

(1)  Intelligence  Systems 

All  Source  Analysis  System 
Automatic  Meteorological  System 
Standoff  Target  Acquisition  System 
Remotely  Monitored  Battlefield  Sensor  System 
Remotely  Piloted  Vehicle 

Electro-Optical  Control  Analysis  and  Reporting  System 
Remote  Automated  Weather  System 
Classified  Interfaces  (8) 

TRAILBLAZER 
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(2)  Electronic  Warfare  Systems 

Multiple  Target  Electronic  Warfare  Systems 
Tactical  Jammer 

Unattended  Expendable  Electronic  Countermeasures 
Close  Air  Support  Electronic  Countermeasures 

(3)  Command  and  Control  Systems 

PATRIOT  CCS/ECS 
Maneuver  and  Control  System 
Nuclear  Burst  Detection  System 
Tactical  Computer  System 
Tactical  Computer  Terminal 
FIREFINDER 

Multiple  Launch  Rocket  System 
SHORAD 

(4)  Communications/Navigation 

Position  Location  Reporting  System- 
Global  Positioning  System 

Joint  Tactical  Information  Distribution  System 
Mobile  Subscriber  Equipment 
Army  Data  Distribution  System 
Source  Data  Automation 

d.  Recommended  TRMS  Priority.  Refer  to  the  preceding  workload  paragraph 
and  the  ODCSOPS  priority  list. 
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e.  Association  with  Requirements  Documentation.  The  Army  Battlefield 
Automation  Interoperability  System  Engineering  Management  Plan  (BAISEMP) , 
dated  November  1978,  outlines  the  requirements  for  interoperability  testing. 
ABIC,  dated  December  1979,  the  draft  ABIC  80,  and  the  ACCS  Implementation  Plan 
provide  further  requirements  details. 
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ABIC 

ACCS 

AIOPTDA 

ATDL 


Array  Battlefield  Interface  Support 
Army  Command  and  Control  System 
Automated  Interoperability  Test  Data  Analysis 
Army  Tactical  Data  Link 


BAISEMP 

BMDP 

CARROTS 

C3I 


Battlefield  Automation  Interoperability  System  Engineering 
Management  Plan 
Biomedical  Data  Package 

Communication  Applications  Real-Time  Remotely  Operable  Test 
System 

Command,  Control,  Communications,  and  Intelligence 


CPCI 

CPU 

C-E 

C&I 


Computer  Program  Configuration  Item 
central  processing  unit 
communications-electronics 
compatibility  and  interoperability 


DBMS 

DOD 

DRA 

DT 


data  base  management  system 
Department  of  Defense 
data  reduction  analysis 
Development  Test 


ECCM 

ECM 

EPR 

FAST 


electronic  counter-countermeasure 
electronic  countermeasure 
equipment  performance  report 
Functional  Area  System  Test 


FATDS 

GDBMS 

HOL 

IEP 


Field  Artillery  Tactical  Data  Systems 
generalized  data  base  management  system 
high  order  language 
Independent  Evaluation  Plan 


INGRES 

ITIS 

JINTACCS 

JITS 


Interactive  Graphics  and  Retrieval  System 
Interim  Test  Item  Stimulator 

Joint  Interoperability  of  Tactical  Command  and  Control 
Systems 

Joint  Interface  Test  System 


JPL 

JTIDS 

MCS 

MFL 


Jet  Propulsion  Laborary 

Joint  Tactical  Information  Distribution  System 
Maneuver  Control  System 
Message  Format  Library 


MLT 

MOP 

MST 

NVVT 


message  log  tape 
measures  of  performance 
message  scenario  tape 
New  Version  Verification  Test 
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PEARS 

PJH 

PLRS 

PTAS 

ROC 

SIR 

SOS 

SSG 

SUT 

TACFIRE 

TADIL 

TAR 

TATERS 

TCAC 

TCS 

TCT 

TDP 

TECOM 

TMDB 

TOE 

TOP 

TPR 

TSSG 

TTY 


Post-Test  Evaluation,  Analysis,  and  Reporting  System 
PLRS  JTIDS  Hybrid 
Position  Location  Reporting  System 
post-test  analysis  system 

Required  Operational  Capabilities 
software  investigation  report 
Standard  Operating  System 
Software  Support  Group 

system  under  test 
Tactical  Fire  Direction  System 
Tactical  Digital  Information  Link 
v  Test  Anomaly  Report 

Tactical  Automated  Test,  Evaluation,  and  Reporting  System 
Tactical  Command  and  Control 
Tactical  Computer  System 
Tactical  Computer  Terminal 

Test  Design  Plan 

US  Army  Test  and  Evaluation  Commend 

test  message  data  base 

Table  of  Organization/Equipment 

Test  Operation  Procedures 
TACFIRE  Problem  Report 
TACFIRE/FATDS  Software  Support  Group 
teletype 


USAEPG 


US  Army  Electronic  Proving  Ground 


APPENDIX  C.  REFERENCES 


1.  Independent  Evaluation  Plan  for  the  Maneuver  Control  System  of  the  Force 
Level  and  Maneuver  Control  (Sigma)  System,  P.  J.  Randall,  April  1981. 

2.  Methodology  Investigation,  Interoperability  Test  Methodology,  H.  V. 
Wallace,  USAEPG,  21  October  1982. 

3.  Engineering  Service  Study  Report,  Application  of  Joint  Interface  Test 
System  (JITS)  Design  to  the  Interim  Test  Item  Stimulator  (ITIS),  by  Command, 
Control,  and  Communications  Corporation,  November  1982. 

4.  Bell  Technical  Operations  Letter,  Subject:  Sample  ITIS  Test  Data, 

19  November  1982. 

5.  Post-Test  Analysis  System  User's  Manual,  USAEPG,  10  December  1982. 

6.  TACFIRE  Field  Manual,  FM  6-1,  TACFIRE  Operations,  Headquarters,  Department 
of  the  Army,  28  September  1979. 

7.  TACFIRE  Reference  Note,  US  Army  Field  Artillery  School,  May  1981. 

8.  TACFIRE  Software  Support  Group,  Test  Plan  for  the  Verification  Testing  of 
Post  Deployment  Versions  of  TACFIRE  Field  Systems,  TACFIRE/FATDS  Software 
Support  Group,  1  April  1980. 

9.  Final  Report,  Development  Test  II  of  Position  Location  Reporting  System, 
CPT  S.  E.  Lord,  December  1981. 

10.  Final  Report,  Supplement  2,  Development  Test  II  of  Position  Location 
Reporting  System,  Software,  CPT  S.  E.  Lord,  September  1982. 

11.  US  Army  TECOM  Test  Operations  Procedure,  Computer/Software  Performance 
Testing,  USAEPG,  1  September  1982. 

12.  Independent  Evaluation  Plan/Test  Design  Plan  (IEP/TDP)  for  the  Tactical 
Computer  Terminal  (TCT)  and  Tactical  Computer  System  (TCS),  L.  F.  Storm,  Head¬ 
quarters,  US  Army  TECOM,  May  1981. 

13.  Force  Level  and  Maneuver  Control  System  Test  Design  Plan,  US  Army  MSAA, 
June  1982. 

14.  Computer  Program  Configuration  Item  Specification,  -Communication  Applica¬ 
tions  Real-Time  Remotely  Operable  Test  System  (CARROTS)  for  Tactical  Automated 
Test,  Evaluation  and  Reporting  System  (TATERS),  USAEPG,  18  July  1979. 

15.  Computer  Program  Configuration  Item  Specification,  Post-Test  Evaluation 
Analysis  and  Reporting  System  (PEARS)  for  Tactical  Automated  Test,  Evaluation, 
and  Reporting  System,  USAEPG,  2  October  1979. 


C-l 


16.  Test  Procedure  2.5.5  -  Message  Prioritization,  OSAEPG. 


17.  Bell  Technical  Operations  Letter,  Subject:  MCS  Scenario  Information, 

16  December  1982. 

18.  Principles  of  Data  Base  Systems,  Second  Edition,  J.  D.  Ullraan,  Computer 
Science  Press,  1982. 


C-2 


APPENDIX  D.  DISTRIBUTION  LIST 


4 


Number 


1 

Addressee 

of  Copies 

Commander 

v' 

US  Army  Test  and  Evaluation  Command 

ATTN:  DRSTE-AD-M 

3 

DRSTE-T0 

2 

Q 


DRSTE-AD-A 

DRSTE-CT 

DRSTE-CM 

Aberdeen  Proving  Ground,  MD 


1 

3 

3 


21005 


V 

y 


I 


Commander 

Defense  Technical  Information  Center 
ATTN:  DTIC-DDR 


2 


w, 

>. 

V, 

> 


v 

V 

V 


V 


Cameron  Station 
Alexandria,  Va  2231': 

Commander 

US  Army  Aberdeen  Proving  Ground 

ATTN:  STEAP-MT-M 

Aberdeen  Proving  Ground,  MD  21005 

Commander 

US  Army  Yuma  Proving  Ground 
ATTN:  STEYP-MSA 
Yuma,  AZ  85364 


2 


2 


i 

-j 

:h 

i  i 

..•I 


i 

.  * 

<*. 

>, 


Commander 

US  Army  Jefferson  Proving  Ground 
ATTN:  STEJP-TD-E 

Madison,  IN  47250 

Commander 

US  Army  Dugvay  Proving  Ground 
ATTN:  STEDP-P0-P 
Dugway,  UT  84022 

Commander 

US  Army  Cold  Regions  Test  Center 
ATTN:  STECR-TM 
AP0  Seattle  98733 


Commander 

US  Army  Electronic  Proving  Ground 

ATTN:  STEEP-MT-T 

Port  Huachuca,  AZ  85613 


1 


1 


1 


4 


.ti 

* 


v 

V 

■} 

U 

U 

X 


D-l 


Addressee 


Commander 

US  Army  Tropic  Teat  Center 
ATTN:  STETC-TD-AB 
APO  Miami  34004 

Commander 

US  Army  White  Sands  Missile  Range 
ATTN:  STEWS-AG-AS-AM  (Record  Copy) 
STEWS-TE-PY 

White  Sands  Missile  Range,  NM  88002 


